ConformalPrediction.jl

Predictive Uncertainty Quantification in Machine Learning

Patrick Altmeyer

Delft University of Technology

July 19, 2023

Conformal Prediction

Conformal Prediction (CP) works under the premise of turning heuristic notions of Predictive Uncertainty (PU) into rigorous ones through repeated sampling or the use of calibration data.

Example: Split CP

  1. Proper training set and separate calibration set: \(\mathcal{D}_n=\mathcal{D}^{\text{train}} \cup \mathcal{D}^{\text{cali}}\).
  2. Train model on proper training set: \(\hat\mu_{i \in \mathcal{D}^{\text{train}}}(X_i,Y_i)\).
  3. Compute nonconformity scores, \(\mathcal{S}\), using calibration data \(\mathcal{D}^{\text{cali}}\) and fitted model \(\hat\mu_{i \in \mathcal{D}^{\text{train}}}\).
  4. For user-specified coverage ratio \((1-\alpha)\) compute the corresponding quantile, \(\hat{q}\), of \(\mathcal{S}\).
  5. For the given quantile and test sample \(X_{\text{test}}\), form the corresponding conformal prediction set: \(C(X_{\text{test}})=\{y:s(X_{\text{test}},y) \le \hat{q}\}\).

Blog posts

Talk Agenda

  1. 🏃 Interactive sprint of ConformalPrediction.jl (15min)
  2. 🔍 Applications (5min)
    • Conformal Chatbot
    • Conformal Image Classifier
  3. 🚧 Under Construction (5min)
  4. ❓ Q&A

🔍 Applications

Conformal Chatbot

Blog post

Building a Conformal Chatbot in Julia ([blog], [TDS])

Figure 1: Demo of a REPL-based conformalized intent classifier.

Conformal Image Classifier

MNIST classifier trained using MLJFlux.jl.

Blog post

How to Conformalize a Deep Image Classifier ([blog], [TDS], [Forem])

🚧 Under Construction

Differentiability

Conformal Bayes

❓ Q&A

Towards Trustworthy AI in Julia

Home of packages geared towards Trustworthy Artificial Intelligence in Julia.

Taija
  1. CounterfactualExplanations.jl (JuliaCon 2022)
  2. ConformalPrediction.jl (JuliaCon 2023)
  3. LaplaceRedudx.jl (JuliaCon 2022)
  4. AlgorithmicRecourseDynamics.jl

… contributions welcome! 😊

📚 More Reading

Image Sources

  • Copyright for stock images belongs to TU Delft.
  • All other images, graphics or animations were created by us.

References